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ABSTRACT 

Error  Growth  In  Poor  ECMWF  Forecasts 
Over  The  Contiguous  United  States.  (August  1993) 

Norman  Ray  Modlin,  B.  S.,  North  Carolina  State  University 
Chair  of  Advisory  Committee:  Dr.  James  P.  McGuirk 

Successive  improvements  to  the  European  Center  for  Medium-range  Weather 
Forecasting  model  have  resulted  in  improved  forecast  performance  over  the  Contiguous 
United  States  (CONUS).  While  the  overall  performance  of  the  model  in  this  region  was 
found  to  have  improved  during  the  period  of  the  1981-1990  winter  seasons,  the  number 
of  poor  forecasts  increased  over  this  time. 

This  study  uses  the  Root  Mean  Square  (RMS)  error  to  measure  the  performance 
of  S-day  500  mb  winter  forecasts  over  the  CONUS.  Poor  and  good  forecasts  were 
defined  in  terms  of  the  10-year  distribution  of  the  RMS  values  between  the  1981  and 
1990  winter  seasons. 

Subjective  analysis  of  a  subset  of  poor  forecasts  yielded  no  obvious  patterns  of 
error  growth,  location  or  propagation  in  the  evolution  of  poor  forecasts.  A  tendency  was 
noted  for  in  situ  amplification  of  forecast  errors.  Additionally,  successive  forecasts 
verifying  on  the  same  day  were  found  to  have  similar  error  patterns,  with  increased 
amplitudes  at  longer  forecast  lengths.  This  implies  that  the  initial  conditions  are  not  a 
significant  source  of  the  error  in  poor  forecasts. 

Empirical  Orthogonal  Function  (EOF)  analysis  of  error  growth  in  time  and  space 
revealed  significant  differences  between  poor  and  good  forecasts.  Good  forecasts  were 
found  to  have  the  majority  of  RMS  growth  on  day  1  while  poor  forecasts  did  not 
experience  rapid  error  growth  until  days  3  and  4.  For  poor  forecasts,  the  leading  EOFs 
revealed  a  wave  pattern  down  stream  of  the  Rocky  Mountains.  This  pattern  evolved  and 
propagated  throughout  the  forecast  period  until  it  dominated  the  S-day  OTor  field.  No 


similar  pattern  was  revealed  in  the  aror  fields  of  good  forecasts.  This  pattern  suggests  a 
dynamic  link  between  the  Rockies  and  the  zonal  wind,  although  no  link  with  the  500  mb 
geostrophic  wind  could  be  established. 
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CHAPTER  I 


INTRODUCTION 

Weather  forecasting  has  been  greatly  advanced  in  the  latter  portion  of  this  century 
due  to  the  advent  of  Numerical  Weather  Prediction  (NWP)  models.  Despite  tremendous 
advances  in  NWP,  the  goal  of  a  perfect  forecast  model  can  not  be  realized  for  many 
reasons.  Among  them  is  the  fiict  that  the  models  use  discrete  observations  and  numerical 
methods  to  analyze  and  predict  continuous  meteorological  fields  and  processes.  As  a 
consequence,  errors  in  these  forecasts  are  an  unpleasant,  yet  unavoidable  result.  By 
studying  the  error  characteristics  of  the  operational  NWP  modds,  we  may  be  able  to 
recognize  in  advance  if  a  model  is  producii^  suspicious  forecast  results.  Additionally,  the 
identification  of  these  characteristics  is  essential  in  order  to  modify  and  improve  their 
performance  over  time.  Knowledge  of  the  shortcomings  of  an  NWP  model,  such  as  the 
European  Center  for  Medium-Range  Weather  Forecasting  (ECMWF)  model  studied  here, 
is  necessary  to  accurately  interpret  the  results  of  the  model  and  to  identify  forecasts  that 
perform  poorly.  Specifically,  this  study  will  investigate  the  growth  of  500  mb  errors  over 
the  Contiguous  United  States,  hereafter  referred  to  as  the  CONUS.  The  focus  of  the  error 
growth  study  will  involve  5-day  forecasts  which  are  identified  as  poor  performers  in  terms 
of  Root  Mean  Square  (RMS)  error  over  this  region.  Comparisons  are  also  made  with 
good  performing  forecasts  at  this  level  and  forecast  length.  The  error  growth  is  studied  in 
both  space  and  time. 

The  objectives  of  this  research  are  presented  in  Ch^er  n.  Chapter  m  discusses 
the  evolution  of  the  forecast  model  and  reviews  previous  studies  of  forecast  model  errors. 
The  data  used  in  this  research  is  described  in  Chapter  IV,  as  well  as  the  procedures  to  be 
used.  In  Chapter  V,  a  climatology  of  error  fields  is  developed,  using  both  the  absolute 


The  style  used  is  that  of  Monthly  Weather  Review. 
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errors  and  the  RMS  error.  Chapter  VI  presents  the  growth  of  the  forecast  errors  in  time 
and  space,  as  well  as  the  sensitivity  of  the  growth  of  errors  to  differing  initial  conditions. 
Finally,  Chapter  Vn  summarizes  all  results  from  the  research  presented.  The  Appendix 
lists  the  valid  dates  of  the  poor  and  good  5-day  forecasts  that  comprise  the  data  sets  used 
in  the  analyses  of  this  study. 
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CHAFTERn 
RESEARCH  OBJECTIVE 

The  objective  of  this  research  is  to  document  the  occurrence  and  temporal  and 
spatial  structure  of  500  mb  forecast  errors  in  poor  S-day  forecasts  over  the  contiguous 
United  States.  The  distribution  of  errors,  both  forecast  and  RMS,  are  investigated  to 
provide  clues  as  to  the  preferred  locations  of  error  occurrence.  Also,  re^onal  difteences 
in  the  long-term  distribution  of  RMS  errors  are  quantified.  The  evolution  of  these  errors 
are  described  in  terms  of  typical  spatial  and  temporal  modes.  It  is  hypothesized  that  there 
are  preferred  modes  or  patterns  for  these  errors  and  that  differences  exist  between  the 
error  modes  of  poor  and  good  fisrecasts.  It  is  also  hypotheazed  that  the  error  patterns 
and  their  growth  are  sensitive  to  the  antecedent  SOO  mb  pattern  used  as  initial  conditions 
for  a  particular  run.  By  documenting  the  existence  and  growth  of  these  error  modes  and 
their  sensitivities,  this  research  identifies  areas  of  deficiency  in  the  ECMWF  model,  as»sts 
in  the  identification  of  poorly  evolving  forecasts  and  provides  input  for  future 
improvements. 

The  objectives  of  this  research  will  be  met  by: 

1 .  The  construction  of  a  climatology  of  the  error  fields. 

2.  A  subjective  study  of  error  solution,  to  include  a  case  study. 

3.  Empirical  Orthogonal  Function  (EOF)  analyas  of  forecast  error  growth  in 
both  time  and  space. 

4.  A  comparison  of  the  EOF  results  for  poor  and  good  forecast  categories. 

5.  A  comparison  of  the  EOF  results  for  differ  time  periods  in  the  evolution 


of  the  ECMWF  model. 


4 


CHAFTERm 

BACKGROUND 

Ever  since  the  pioneering  attempt  at  NWP  by  Richardson  (1922),  forecasts  have 
been  verified  against  the  existing  analyses  to  gauge  the  accuracy  of  the  prediction  and  to 
identify  and  correct  areas  of  deficiency.  Retrospective  looks  such  as  Platzman  (1967)  and 
Lynch  (1992)  have  shown  that  Richardson’s  model  could  not  overcome  problems  with  the 
data  used  as  initial  conditions.  Recognition  of  such  limitations  has  resulted  in  successivdy 
improved  models,  up  through  the  current  ECMWF  model.  As  with  previous  models,  this 
one  has  been  studied  repeatedly  to  determine  its  error  characteristics. 

Analysis  of  the  error  characteristics  of  the  ECMWF  must  be  viewed  in  light  of  the 
changes  made  to  the  model  over  time,  which  are  intended  to  reduce  the  total  fixecast 
error.  These  changes  are  documented  by  Trenberth  and  Olson  (1988).  The  most 
significant  changes  include  the  following: 

a.  Change  fi'om  a  grid-point  to  T63  spectral  model  in  April  1983. 

b.  Inclusion  of  envelope  orognq}lQr  in  February  1984. 

c.  Extensive  change  to  the  model’s  analysis  procedures  in  May  1984. 

d.  Modifications  to  radiative  parameterization  in  December  1984. 

e.  Increase  in  horizontal  resolution  to  T106  and  modification  to  parameterizations 
of  convection,  clouds  and  condensation  in  May  1985. 

f  Increased  vertical  resolution  fi’om  16  to  19  levels  in  May  1986. 

g.  Revised  use  of  observations  and  data  selection  methods  which  improved  the 
analysis  procedure  in  SqMember  1986. 

Lorenz  (1982)  performed  a  study  of  500  mb  height  forecasts  versus  analytis  uting 
data  from  the  1980-81  winter  season  operational  ECMWF  grid-point  modd.  This  dudy 
analyzed  the  global  root  mean  square  (RMS)  errors  in  the  forecast  verification  uting  the 
height  fields  in  spectral  format.  He  noted  that  we  have  a  good  undostanding  of  the 
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dynamical  processes  wdiich  govern  atmospheric  motions,  wdiich  is  essential  in  (»der  to 
predict  the  atmosphere’s  future  state.  Howevo’,  the  atmosphere  itsdf  is  unstable  with 
respect  to  small-scale  perturbations.  Tho’efore,  observations  which  accurately  measure 
large-scale  features,  but  contain  uncertainties  at  smaller  scales,  will  lead  to  forecasts 
containing  large-scale  errors  comparable  to  those  wdiich  would  exist  if  large-scale  errors 
were  present  in  the  initial  data. 

Lorenz  (1989)  studied  simulations  of  a  simple  chaotic  forecast  system  in  an  eflfort 
to  determine  which  has  the  greater  effect  on  overall  system  performance;  improvements  to 
the  analysis  procedures  or  improvements  to  the  forecast  model.  The  simulation  forecast 
model  used  sets  of  equations  for  which  the  solution  could  be  computed  and  used  as  exact 
initial  data.  Analysis  and  model  errors  were  then  added  to  the  data  and  forecast  system, 
respectively,  and  then  reduced  singly  and  in  combination.  Improvements  to  the  analyas 
procedure  alone  were  found  to  bring  the  analy»s  more  in  line  with  the  initial  data,  while 
increasing  the  difference  between  the  analysis  and  forecasts.  The  analysis  improvements 
were  found  to  be  most  effective  at  shorter  time  ranges.  Forecast  model  improvements, 
when  viewed  alone,  reduced  the  total  error  as  compared  to  the  initial  data,  while  reducing 
the  forecast  to  analysis  difference.  Model  improvements  were  most  effective  at  longer 
time  ranges.  The  greatest  benefits  to  the  overall  system  were  achieved  in  concert,  that  is 
model  improvements  were  more  effective  if  the  analysis  had  been  previously  improved, 
and  vice  versa.  While  these  results  may  seem  intuitive,  Lorenz  (1989)  states  that  the 
results  may  not  directly  apply  to  the  ECMWF  system,  because  the  simulations  were  not 
based  on  real  atmospheric  behavior  and  since  daily  ECMWF  error  behavior  may  not  be 
independent  of  preceding  days  or  the  initial  state. 

The  systematic  error  characteristics  of  the  spectral  ECMWF  were  reported  by 
Arpe  and  Klinker  (1986).  The  spectral  model  was  chosen  to  replace  the  grid-point  modd 
because  the  spectral  model  more  accurately  predicts  the  movement  of  short  waves  (Girard 
and  Jarraud  1982)  vdiile  achieving  greater  computational  efBciency  (Simmons  1983).  For 
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the  period  April  1983  -  April  1985,  Arpe  and  Klinker  (1986)  found  that  the  spectral  modd 
correctly  identified  the  positions  of  500  mb  troughs  in  10-day  forecasts,  but  incorrectly 
forecast  their  amplitudes.  The  errors  exhibit  a  wave  number  2  pattern  with  500  mb 
heights  under  forecast  over  Northon  Europe  and  the  Bering  Strait,  and  hei^s  ova* 
forecast  over  Canada  and  Siberia.  This  error  pattern  was  found  to  devdop  fix>m  the  onset 
of  the  10-day  forecast  period,  with  the  error  growing  in  place  continually  through  the 
period.  An  error  dipole  pattern  was  found  to  exist  near  the  main  mcxiiitain  ranges.  This 
dipole  continued,  although  reduced  in  amplitude,  after  the  introduction  of  envdope 
orography  in  February  1984.  Arpe  and  Klinker  (1986),  as  wdl  as  Arpe  (1989),  fixind  that 
the  error  fields  exhibited  an  equivalent  barotropic  characteristic,  although  the  error 
amplitudes  did  increase  with  hdght.  Additionally,  Arpe  (1989)  found  that  the  most 
significant  reductions  of  systematic  error  were  achieved  by  modifications  to  the  convective 
and  radiative  parameterizations  in  Decend>er  1984  and  May  1985,  respectively,  with  the 
most  significant  improvements  noted  in  the  winter  season.  This  study  also  noted  that  the 
ECMWF  model  has  poor  skill  in  predicting  blocking  rituations. 

The  work  of  Lorenz  (1982)  was  extended  by  Dalcher  and  Kalnay  (1987).  Their 
study  separated  total  forecast  mean  square  error  into  systematic  and  random  componrats. 
This  work  found  that  systematic  errors  were  dominated  by  low  total  wave  numbers  and 
that  the  error  growth  rate  increases  monotonically  with  total  wave  number.  Dalcher  and 
Kalnay  (1987)  also  advocate  the  measurement  of  error  growth  at  finite  times  versus  the 
Lorenz  (1982)  concept  of  "doubling  time  of  snail  errors,”  since  the  determination  of  snudl 
errors  can  lead  to  extrapolation  difiSculties  in  a  discrete  data  set.  The  current  study  is 
concerned  with  error  growth  measured  at  finite  times. 

A  detailed  study  of  ihe  ECMWF  model  performance  over  the  CONUS  was 
accomplished  by  Netterville  (1991).  This  study  identified  specific  ECMWF  5-day 
forecasts  of  500  mb  height  which  performed  poorly  over  the  CONUS.  The  period  of 
study  was  the  1981-1990  winter  seasons.  Poor  forecast  performances  were  identified  in 
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terms  of  the  RMS  error  of  the  forecast  over  the  CONUS,  and  its  rdadon  to  the  annual 
distribution  of  these  RMS  values.  The  significant  finding  was  that  aU  cases  of  poor  S-day 
forecasts  were  found  to  have  anonudous  height  departures  at  the  initial  time  of  the 
forecast  in  one  of  the  three  known  Northern  Hemisphere  blocking  regions.  However,  the 
presence  and  recognition  of  the  blocking  pattmis  alone  was  judged  iiuulequate  to 
determine  a  priori  the  skill  of  individual  5-day  ECMWF  forecast  over  the  CONUS. 
Additionally,  Netterville  (1991)  noted  that  both  tlw  winter  season  S-day  mean  RMS  error 
and  its  variability  over  the  CONUS  region  decreased  over  the  period  of  study,  indicating 
increased  forecast  performance.  However  the  annual  number  of  poor  5-day  forecasts 
increased  toward  the  end  of  the  period  of  study.  This  result  indicates  that  the  distribution 
of  RMS  errors  at  5  days  may  not  be  normal,  but  instead  may  be  skewed  toward  largo* 
values.  However,  this  issue  was  not  addressed  directly  by  Netterville  (1991). 

In  summary,  much  work  has  been  accomplished  to  document  the  systematic  oror 
characteristics  of  the  ECMWF  model.  As  hematic  errors  were  identified,  successive 
improvements  to  the  model  were  incorporated  in  order  to  reduce  these  errors.  These 
improved  models  have  also  been  studied  to  document  the  effects  of  the  changes.  Error 
evolution  has  been  studied  both  theoretically  and  empirically,  but  on  a  global  basis. 
Evolution  of  errors  over  a  limited  region,  spedfically  for  q^ecific  classes  of  forecast 
performance,  has  not  been  presented. 

This  study  will  incorporate  the  results  fi’om  these  previous  studies,  most  notably 
those  of  Netterville  (1991),  in  order  to  measure  the  time  evolution  of  the  error  fidd  for 
poor  forecast  performances,  its  deviation  from  known  systematic  errors  and  atmo^heric 
modes,  as  well  as  the  sentitivity  of  the  error  evolution  to  initial  conditions.  However, 
while  Netterville  (1991)  focused  on  the  rdationship  between  initial  conditions  and  frrrecast 
performance,  the  research  presented  here  will  study  evolutionary  fiictors  between  the  time 
of  the  initial  conditions  and  the  resulting  5-day  forecast. 
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CHAFFER  IV 
DATA  AND  PROCEDURES 

A.  Data 

The  data  set  used  in  this  study  consists  of  SOO  mb  analyses  and  forecasts  for  tiie 
1981-1990  winter  seasons  based  on  the  global  operational  ECMWF  modd.  A  winto* 
season  is  defined  as  a  100  day  period  be^nning  on  1  Dec  of  tl»  previous  year;  that  is,  1 
Dec  1980  starts  the  100  days  of  the  1981  winter  season.  For  each  of  the  100  days,  the 
verifying  analysis  for  00  UTC  is  available,  as  well  as  the  ten  forecasts  for  this  valid  time 
which  were  initiated  1-10  days  prior.  This  format  is  commonly  refisrred  to  as  the  "Lorenz 
format"  and  is  further  described  by  Lorenz  (1982).  The  data  are  archived  on  magnetic 
tape  as  global  coeffidents  of  ^herical  harmonics  with  triangular  truncation  at  total  wave 
number  40.  TIm  spectral  coeffidents  were  converted  to  conventional  heights  (meters) 
usii^  routines  obtained  fi-om  ECMWF  and  nuqjped  onto  a  3.7S  by  3.7S  d^ree  grid  over 
the  Northern  Hemisphere  in  this  study  for  ease  of  computation.  The  resulting  grid  pdnts 
over  the  Northern  Hemiqshere  are  shown  in  Fig.  1.  In  keeping  with  Netterville  (1991), 
the  region  of  interest  is  the  CONUS  region  bounded  by  60  W,  131.25  W,  22.S  N  and 
56.25  N.  This  r^on  with  correspondii^  grid  pdnts  is  shown  in  Fig.  2. 

B.  Procedures 

Forecast  performance  at  5-days  over  the  CONUS  is  judged  in  terms  of  the  RMS 
statistic.  Forecast  performance  will  be  analyzed  udng  the  following  tadcs; 

a.  Calculate  the  absolute  errors  ova*  the  Northern  Hemi^here  fisr  the  entire  data 
set. 

b.  Compute  the  RMS  value  for  the  CONUS  for  all  forecasts. 

c.  Construct  10-year  histograms  of  the  RMS  values  and  compute  the  basic 


statistics  of  the  distribution. 


Figure  1.  Northern  Hemisphere  with  3.75  by  3.75  degree  grid. 


Figure  2.  CONUS  region  of  interest  with  3.75  by  3.75  degree  grid. 
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d.  Define  critical  values  for  poor  and  good  forecasts  based  on  the  10-year 
distribution  of  RMS  values. 

The  distribution  of  RMS  error  over  the  CONUS  will  be  compared  with  otha* 
regions  of  the  Northern  Hemisphere.  This  re^onal  variation  in  forecast  poformance  will 
be  studied  in  the  following  manner; 

a.  Perform  steps  b.  -  d.  above  for  the  other  selected  region. 

b.  Perform  statistical  tests  for  significance  between  the  RMS  distributions  of  the 
two  regions. 

Poor  forecast  error  growth  through  time  will  be  diagnosed  using  the  following 
procedures; 

a.  A  sample  of  poor  forecasts  will  be  examined  subjectivdy  for  the  origin,  growth 
and  location  of  error  patterns. 

b.  The  sample  set  will  be  examined  for  differences  in  error  fields  of  poor  forecasts 
produced  by  initial  conditions  of  na^jboring  days. 

c.  Significant  results  and  patterns,  if  any,  fi-om  a.  and  b.  will  be  examined 
throughout  the  entire  data  set. 

d.  Empirical  Orthogonal  Function  (EOF)  analysis  will  be  performed  over  time  on 
the  RMS  values  of  forecast  evolution  for  both  good  and  poor  forecast  cases. 

Spatial  patterns  of  error  growth  will  be  examined  by  the  use  of  EOF  analyins.  Two 
types  of  EOF  analysis  will  be  performed.  These  analyses  will  be  performed  on  the 
correlation  and  covariance  matrices  of  the  error  fidds.  Additionally,  both  EOF  analyses 
are  performed  with  and  without  the  mean  error  fidd.  The  correlation  EOF  analysis  is  used 
to  extract  error  modes  >^ch  are  smoothed  based  on  the  ^andard  deviation  of  each  data 
point.  The  correlation  EOFs  accent  areas  of  high  variability,  because  they  are  not 
smoothed  about  the  standard  deviations.  By  leaving  in  the  mean  error  fidd,  the  EOFs 
represent  deviations  about  the  mean  error.  However,  by  removing  or  correcting  for  the 
mean  error,  the  dominant  EOF  is  the  mean  error  field  and  tl^  remaining  EOFs  are 


subordinate  to  the  mean  pattern.  The  EOF  analysis  will  be  accomplished  in  the  following 
manner; 

a.  Perform  EOF  analysis  on  the  error  fields  of  poor  and  good  forecasts. 

b.  Examine  any  differences  which  exist  between  these  analyses. 

The  posable  effects  of  bias  in  the  case  selection,  as  well  as  the  effects  of  successive 
model  improvements  will  be  analyzed  as  follows; 

a.  Segregate  the  poor  forecasts  based  on  the  operational  vosion  of  the  model 
which  was  used  to  produce  them. 

b.  Perform  EOF  analysis  on  the  error  fields  of  these  new  data  sets. 

c.  Compare  the  modes  revealed  fi*om  each  version  of  the  model  with  each  other 
and  with  the  modes  revealed  in  the  10  year  analysis. 

A  possible  cause  of  the  poor  forecasts  will  be  investigated  by  the  following; 

a.  Compute  the  mean  SOO  mb  zonal  wind  speeds  over  different  geographical 
regions  for  the  length  of  the  forecast  period. 

b.  Plot  the  distribution  of  the  mean  zonal  wind  speeds  against  the  corresponding 
5-day  RMS  values. 

c.  Examine  the  differences  between  the  distributions  of  the  zonal  wind  speeds  for 
poor  and  good  forecasts. 

C.  Equations 

Throughout  the  course  of  this  study,  several  equations  and  statistical  tests  will  be 
used.  They  are  summarized  below. 

In  keeping  with  Netterville  (1991),  the  RMS  error  was  chosoi  as  the  standard  of 
performance  used  to  judge  forecasts.  The  familiar  formula  for  this  measure  is  as  follows; 

RMS  =  \-'t(x,-X,ff  (1) 

L"  1  J 

where  =  forecast  height  at  a  grid  point,  X„  =  observed  height  at  a  grid  point  and  n  = 
number  of  grid  points  in  the  region  of  interest. 
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The  goodness-of-fit  test  (Ott  1988)  is  used  to  determine  the  statistical 


significance  of  a  distribution  across  several  categories.  The  formula  for  this  statistical  test 
is; 


(2) 


where  /  =  the  number  of  cat^ories,  -  the  observed  number  in  category  i  and  =  the 
expected  number  in  category  i  .  The  null  hypothecs  for  this  test  is  that  all  probabilities 
for  each  category  are  equal  to  a  specified  s^  of  probabilities.  The  nuU  hypothec  is 
rejected  if  exceeds  a  critical  value  which  is  based  on  the  number  of  categories  and  the 
desired  level  of  confidence. 

The  geostrophic  wind  equation  is  used  to  compute  a  value  of  the  mean  zonal  wind 


over  the  region  of  interest.  The  form  of  this  equation  fi-om  Hohon  (1993)  is; 

1  ^ 

u  =  -■ 


f  4^ 


(3) 


where  u  is  the  mean  zonal  wind,  /  is  the  Coriolis  parameter  and  ^  is  the  gradient  of 

4' 


mean  SOO  mb  geopotential  heights  over  a  spedfied  latitudinal  band.  The  geopotential 
heights  are  computed  fi'om  the  500  mb  height  data. 
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CHAPTER  V 

DISTRIBUTION  OF  ERRORS 

Prior  to  determining  patterns  of  error  growth,  it  was  decided  to  look  at  the 
distributions  of  the  actual  forecast  errors.  This  was  done  to  determine  the  rdative 
importance  of  the  CONUS  within  the  overall  distribution  of  errors.  Additionally,  the 
distribution  of  the  CONUS  RMS  values  was  revisited  to  study  its  statistical  characteristics 
and  seek  improved  criteria  for  defining  poor  forecasts.  Finally,  the  distribution  of  RMS 
error  values  over  a  portion  of  the  Pacific  Ocean  was  studied  to  determine  if  dififovnces 
exist  between  geographical  regions. 

A.  Distribution  of  forecast  errors 

Since  the  critical  error  values  used  by  Netterville  (1991)  are  on  the  order  of  100 
meters  for  S-day  forecasts,  this  number  is  used  as  a  representative  value  to  measure  the 
distribution  of  errors.  For  each  grid  point  over  the  Northern  Hemisphere,  counts  were 
made  of  the  number  of  days  with  100  meter  errors  of  each  sign,  as  well  as  for  the  absolute 
value  of  the  error.  These  fi-equency  plots  for  the  5-day  forecast  are  depicted  in  Fig.  3. 

Several  features  are  apparent  upon  inspection  of  Fig.  3a.  Most  notable  are  the 
maxima  located  near  Newfoundland  and  in  tl»  Southern  Gulf  of  Alaska.  These  maxima 
affect  the  extreme  northeast  and  northwest  comers,  respectively,  of  the  CONUS  area  of 
interest.  The  CONUS  region  itsdf  lies  in  a  rdative  minimum  of  absolute  error  fi-equency. 
These  two  maxima  corre^nd  to  previously  documented  climatological  r^ons  of 
explosive  cyclogenesis  (Roebber  1984,  Uccellini  1990,  Alberta  et.  al.  1991).  The 
maximum  over  Newfoundland  is  dominated  by  positive  forecast  errors  (Fig.  3b),  ^ch 
indicates  that  the  main  problem  for  this  r^on  is  the  over  forecast  of  SOO  mb  hei^s. 
Another  positive  maximum  is  present  over  central  Canada,  which  corresponds  to  the  result 
of  Arpe  and  Klinker  (1986).  Conversely,  the  Gulf  of  Alaska  maximum  is  dominated  by 
negative  errors,  (Fig  3c),  indicating  the  preference  for  over  development  of  cyclones  or 
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Figures.  (Continued). 
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Figures.  (Continued). 
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underdevelopment  of  anticyclones  in  the  region.  This  mgative  maximum  is  sli^tly 
displaced  eastward  from  the  result  of  Arpe  and  Klinker  (1986).  The  regions  of  maximum 
positive  errors  over  the  Pacific  Ocean  and  northeastern  North  America  and  n^ative  errors 
over  the  Gulf  of  Alaska  are  aligned  with  the  dominant  storm  tracks  in  these  regions.  Over 
the  hemisphere,  the  steepest  gradient  of  this  error  frequency  distribution  also  occurs 
through  the  mid-latitudes,  the  region  of  transient,  synoptic  scale  cyclones.  Conversely,  the 
distributions  in  the  tropics  and  polar  regions  are  relatively  uniform,  owing  to  the  low 
frequency  dynamics  of  these  regions.  Together,  these  results  indicate  that  the  most 
important  r^ons  of  error  are  collocated  with  the  regions  of  greatest  activity  or  variability 
in  the  atmosphere. 

Within  the  CONUS,  the  oror  frequency  pattern  is  relatively  zonal,  especially  in  the 
Southeast,  and  increases  poleward.  However,  a  localized  minima  exists  along  and  in  the 
lee  of  the  Rocky  Mountains,  primarily  in  the  negative  phase  (Fig.  3c).  This  result  shows 
that  cyclones  are  forecast  well  in  this  region,  even  though  the  region  is  a  preferred  location 
for  maximum  cyclonic  development  (Rod>ber  1984).  However,  this  performance  may  be 
enhanced  by  noting  that  little  if  any  winter  season  explosive  cyclogenesis  occurs  in  the 
region  (Roebber  1984). 

B.  Distribution  of  RMS  errors 

Netterrille  (1991)  presented  time  series  of  the  RMS  values  over  the  CONUS  to 
show  their  daily  variability.  However,  he  did  not  discuss  the  overall  distribution  of  these 
values  with  respect  to  the  critical  values  which  he  used  to  define  a  poor  forecast.  Ifis  use 
of  the  annual  mean  RMS  plus  one  standard  deviation  for  that  year,  while  attempting  to 
allow  for  model  changes  and  improvements,  created  something  of  a  paradox.  The  annual 
mean  RMS  values,  as  well  as  its  annual  variability  (as  measured  by  the  standard  deviation), 
generally  decreased  over  the  10-year  period,  indicating  improved  modd  performance. 
Concurrently,  the  number  of  cases  which  he  defined  as  poor  increased  in  the  latter  half  of 
the  period  of  investigation  relative  to  the  first  half  of  the  period.  The  resulting  paradox  is 
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that,  as  the  model  performs  better  overall,  the  number  of  "poor  performances"  increases. 
This  increase  in  the  number  of  poor  forecasts  is  with  respect  to  a  decreasing  criterion 
(mean  RMS  error)  defining  poor  forecasts.  Thus  what  Netterville  (1991)  actually  shows 
is  that  as  time  has  passed,  the  number  of  forecasts  with  RMS  errors  greater  than  one 
standard  deviation  (for  that  season)  has  increased.  Thus  the  distribution  of  forecast  errors 
about  the  seasonal  mean  has  become  slightly  more  skewed  with  time. 

The  annual  histograms  of  RMS  values  (not  shown)  were  studied  for  their  statistical 
characteristics.  None  of  the  years  revealed  any  significant  bimodal  distributions  and  the 
critical  values  used  by  Netterville  (1991)  were  not  located  within  any  local  maxima  in  the 
distribution.  Thus  his  criteria  for  the  definition  of  poor  forecasts  is  rather  arbitrary.  The 
10-year  composite  distribution  for  day  5  (Fig.  4)  shows  a  near  normal  distribution,  albeit 
slightly  skewed  toward  the  higher  RMS  values. 

In  the  interest  of  judging  all  forecasts  by  the  same  criteria,  poor  forecasts  were 
redefined  in  terms  of  the  lO-year  RMS  statistics.  For  later  comparisons,  poor  and  good 
forecast  RMS  critical  values  were  defined  to  be  the  10-year  mean  RMS  plus  and  minus  the 
10-year  standard  deviation,  respectively.  Hus  yields  critical  values  of  RMS  values  greater 
than  104.49  m  for  poor  forecasts  and  less  than  56.40  m  for  good  forecasts.  The  annual 
number  of  poor  forecasts  uring  this  criteria  is  shown  in  Table  1,  while  the  distribution  for 
good  forecasts  is  shown  in  Table  2.  Using  this  criteria  for  the  selection  of  cases  removes 
the  aforementioned  paradox,  i.  e.  ,  the  number  of  poor  forecasts  generally  decreases  with 
time  just  as  the  RMS  measure  of  performance  also  decreases.  This  criterion  for  poor  and 
good  forecasts  was  used  to  categorize  forecasts  for  this  research.  One  result  of  this 
definition  is  that  the  set  of  poor  forecasts  are  biased  towards  the  earlier  years  of  the  data 
set.  Conversely,  the  good  forecasts  will  be  skewed  toward  the  later  years  of  the  period. 
Notably,  the  earlier  period  contains  the  grid-point  and  T63  spectral  models,  while  the 
latter  half  of  the  record  was  produced  by  the  higher  resolution  T106  spectral  model 
(Trenberth  and  Olson  1988).  Additionally,  the  modifications  to  the  radiative,  convective. 
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cloud  and  condensation  parameterizations  introduced  in  1984  and  1985  were  fcHind  to  be 
the  most  significant  in  terms  of  reducing  the  systematic  error  of  the  modd  (Arpe  1989). 
The  differences  of  these  two  data  sets  will  be  considered  when  investigating  the 
differences  between  the  two  categories. 

The  RMS  values  were  also  computed  over  a  r^on  of  equal  aze  over  the  Padfic 
Ocean.  The  region  is  bounded  by  the  sanw  latitudes  as  the  CONUS  r^on  (22.S  •  56.25 
degrees  N)  and  by  longitudes  135  W  and  153.75  E.  The  10-year  frequency  distribution  of 
these  RMS  errors  is  shown  in  Fig..  5.  Poor  and  good  forecasts  were  defined  in  the  same 
manner  as  over  the  CONUS  (regional  10-year  mean  RMS  plus  or  minus  one  standard 
deviation).  This  yielded  critical  values  of  105.1  Im  (59.19m)  for  poor  (good)  forecasts. 
The  annual  distributions  of  the  poor  and  good  forecasts  for  the  Pacific  region  are  also 
contained  in  Tables  1  and  2.  For  the  Pacific  re^on,  151  poor  forecast  cases  and  36  good 
forecast  cases  were  identified.  As  with  the  distribution  over  the  CONUS,  the  Pacific 
region  distribution  is  rdatively  normal  and  skewed  slightly  toward  higher  values. 

The  RMS  errors  were  computed  ov^-  both  geographical  regions  to  see  if  r^onal 
difierences  exist  in  their  distribution.  The  nuU  hypothesis  that  the  two  distributions  are 
different  was  tested  using  the  Student's  two-tailed  t-test  with  unequal  variances  (Ott 
1988).  The  resulting  p-vahie  fi^om  this  test  is  greater  than  .1,  indicating  that  there  is  a 
greater  than  1  in  10  chance  of  sdecting  the  two  distributions  at  random.  Thertfore,  thoe 
is  insufiSdent  evidence  to  rqect  the  nuU  hypothecs.  Failure  to  reject  the  nuU  hypothesis 
lends  support  to  the  assumption  the  forecasts  for  the  CONUS  and  the  Pacific  do  not 
present  different  problems,  at  least  when  viewed  fi-om  an  RMS  standpoint.  Because  the 
two  regions  are  seemingly  dmilar,  no  fiirther  interest  was  expressed  in  the  errors  ova*  the 
Pacific  region  and  the  remainder  of  this  research  will  deal  with  the  errors  over  the 
CONUS. 
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Table  1.  Annual  distribution  of  poor  5-day  forecast  cases  for  the  CONUS  and  Pacific 
regions.  See  text  for  definition  of  poor  forecasts. 
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CHAPTER  VI 

GROWTH  OF  FORECAST  ERRORS 

Based  <m  the  separatioo  of  die  ECMWF  forecasts  into  poor  and  good  categories 
as  measured  by  the  S-day  RMS  error  ovar  die  CONUS,  the  growdi  of  forecast  errors 
will  now  be  examined.  This  examination  will  reveal  the  dominant  modes  of  error 
growth,  bodi  in  time  and  space.  Addititmally,  it  will  reveal  differences  between  die 
dominant  modes  of  poOT  and  good  forecasts.  This  chapter  piesoits  die  methods  used  in 
this  search  and  the  results  of  die  examination. 

A.  Growth  from  initial  conditions 

Plots  of  the  errOT  fields  were  scremied  subjectively  to  identify  any  patterns  diat 
might  exist  in  die  evoluticm  in  die  fields.  Initially,  the  S-day  forecast  error  fields  were 
studied  and  classified  subjectively  by  patterns  in  these  fields.  However,  it  was  found 
that  no  dominant  pattern  existed  which  provided  useful  informatioo  about  die  error 
growth.  Pattmns  determined  subjectively  ranged  from  error  d^les  over  die  CONUS 
to  hemiqdieric  error  trains.  These  error  dipoles  exhibited  no  preferred  orientation. 

It  was  also  decided  to  trace  the  growdi  of  the  error  fields  dirough  time  frmn  the 
initial  condititms.  Nettmrille  (1991)  found  that  the  initial  conditions  by  diemselves 
were  not  an  adequate  {Hedictcn-  of  fcnecast  skill.  With  diis  study,  it  was  hoped  to  (tetect 
some  kind  of  transient  feature  adiich  produces  a  large  error  field  and  the  resulting  large 
RMS  value. 

To  acconqilish  diis,  a  subset  of  the  poOT  forecast  cases  was  studied.  From  the 
135  poor  5-day  CONUS  forecasts  identified  in  die  previous  chapter,  18  woe  sdected  at 
random.  TIk  verifying  500  mb  analysis  and  ctmeqxmding  5-day  5(X)  mb  forecast  for 
(me  of  die  cases  is  shown  in  Fig.  6.  This  particular  forecast  exhibits  several  common 
forecast  problems.  For  instance,  die  5-day  forecast  (Fig  6b.)  attenqits  to  move  die 
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cutoff  low  soudiwest  of  Baja  CalifcHuia  too  far  eastward.  Additionally,  the  5-day 
fmecast  exhibits  a  zonal  flow  over  the  Norttem  CONUS,  v«sus  the  verifying 
mmhwesterly  flow  ova  the  northwestern  CONUS.  The  S-day  forecast  also  incorrectly 
placed  a  high  pressure  centtf  ova*  Florida  as  well  as  over  developing  the  high  pressure 
center  nmlh  of  die  Aloidans. 

Using  analogous  definitions  (10-year  mean  plus  standard  deviation  of  RMS),  the 
forecasts  for  days  1-4  were  judged  fn-  their  poformance.  Of  the  135  poor  5-day 
CONUS  forecast  cases,  the  forecasts  fOT  days  1-4  were  also  judged  to  be  poor  25.8, 
34.4,  39.1  and  57.8  percent  of  the  time,  reflectively.  These  results  indicate  that  the 
greatest  proportion  (42.2%)  of  5-day  fnecasts  become  poor  on  the  fifth  day. 
Additionally,  it  is  interesting  to  note  that  one  fourdi  of  the  pocx*  5-day  forecasts  are 
already  poor  forecasts  on  the  first  day  of  evoludcm.  It  also  shows  diat  poor  5-day 
forecasts  tend  to  become  poor  at  some  point  in  die  middle,  rather  than  near  the 
beginning,  of  its  evolution.  This  result  has  also  been  documented  by  Lyons  (1992). 

A  case  study  of  error  field  evolutiai  is  shown  in  Fig.  7.  This  figure  shows  die 
daily  evolution  of  the  error  field  as  forecast  from  the  initial  conditions  which  produced 
the  poor  5-day  CONUS  forecast  valid  8  Jan  82.  This  valid  dale  is  the  same  as  die  case 
shown  in  Fig.  6.  This  particular  set  of  initial  conditions  jntiduced  poor  forecasts  (m 
days  1,  2,  4,  5  and  6.  The  main  forecast  {nobtems,  or  dominant  otots,  become 
apparent  in  this  case  by  the  diird  and  fourth  days.  By  day  3,  the  regimi  of  negative 
errors,  or  under  forecast  heights  associated  with  the  premature  eastward  movemrat  of 
the  low  pressure  coiter  is  a^iarent  over  Southern  and  Baja  California.  Likewise,  so 
are  the  positive  errors,  or  over  forecast  heights,  fiirdier  to  die  west  in  die  regicm  whoe 
the  low  pressure  actually  should  be.  A  neptive/positive/negative  arm*  pattern 
stretching  from  the  Al^itians  to  the  Canadian-United  States  border  becomes  most 
apparoit  between  days  3  and  4.  This  error  train  is  associated  with  inccnrecdy  forecast 


Figure  7.  Daily  forecast  errors  evolving  from  the  initial  conditions  which  produce  a 
poor  5-day  forecast  valid  8  Jan  82.  (a.)  Day  1,  (b.)  day  2,  (c.)  day  3,  (d.)  day  4,  (e.) 
day  5  and  (f.)  day  6  errors.  Contour  int^vals  are  (a.),  (b.)  and  (c.)  50  meters  and 
(d.),  (e.)  and  (f.)  100  meters.  Negative  contours  are  dashed. 
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high  pressure  cotter  north  of  the  Aleutians  and  die  ntxdiwestaiy  versus  zonal  flow 
along  the  Canadian-United  States  Ixuder. 

Two  possibilities  exist  in  the  evolution  of  the  etnx  fields.  The  first  is  that  the 
errors  ovo  the  CONUS  would  result  from  a  transient  pattern  of  error  growth.  Such  a 
transient  feature  might  originate  fiom  a  geographical  regirm  ot  regirms  which  are 
poorly  sanqiled  by  the  ground-based  diservational  network,  such  as  the  tropical  East 
Pacific  Ocean.  An  alternative  result  would  be  to  find  diat  die  ertOT  patterns  grow 
largely  in  place  as  previously  documoited  by  Arpe  and  Klinker  (1986). 

In  this  case  study,  as  in  most  of  diis  subset,  the  error  fields  were  found  to 
amplify  in  situ  rather  dian  as  a  result  of  any  transient  feature.  No  obvious  growth  was 
found  to  move  out  of  the  tnqncal  East  Pacific.  However,  there  were  some  cases  niiich 
exhibited  prominent  errors  in  this  region.  In  fiict,  even  though  the  poor  forecasts  were 
defined  based  on  performance  over  the  CONUS,  there  were  many  instances  through  die 
entire  data  set  in  which  the  maximum  error  amplitude  was  found  well  outside  the 
CONUS  region.  No  correlation  can  be  made  between  these  distant  error  maxima  and 
the  error  patterns  over  the  CONUS,  as  dieir  positions  spanned  the  entire  NOTthem 
Hemisphere  mid-latitudes.  Likewise,  they  were  found  to  occur  bodi  as  mid-latitude 
error  wave  trains  and  as  stand  alone  features. 

The  study  of  die  subset  of  poor  forecasts  did  not  produce  any  “smiting  gun” 
pattern  of  error  growth.  The  main  conclusion  was  diat  the  eventual  oror  pattern  of  die 
poor  5-day  forecasts  grows  in  place  with  the  bulk  of  the  growdi  occurring  aftm’  the 
third  or  fourth  day  of  the  forecast.  Additicmally,  it  does  not  appear  diat  the  mrors  are 
the  result  of  any  transient  feature  or  one  that  originates  in  a  poorly  sanqiled  region. 
Because  of  the  limited  results  of  this  study,  it  was  not  applied  to  die  mitire  catalog  of 
poor  forecasts  and  other  avmiues  of  research  were  followed. 
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3.  Sensitivity  to  initial  conditions 

In  addition  to  the  study  of  error  growth  described  above,  die  sensitivity  of  error 
growth  of  the  ECMWF  model  to  the  initial  conditions  was  examined.  To  acconqilish 
this,  the  same  subset  of  poor  5-day  forecast  dates  was  used  and  the  4-  and  6-day 
forecast  error  fields  were  studied.  These  fOTe.:asts  were  also  valid  on  the  same  date  as 
the  poor  S-day  forecasts.  This  study  was  performed  to  determine  if  the  performance  of 
the  ECMWF  is  sensitive  to  the  data  used  as  initial  conditions  for  the  model.  The 
underlying  hypothesis  is  that  the  forecast  fm*  a  given  date  will  improve  as  the  fmecast 
length  is  shortened  and  worsen  as  the  forecast  length  is  lengthened.  More  specifically, 
if  the  initial  conditions  are  important,  the  likelihood  is  small  that  the  initial  emvs  will 
be  the  same  for  three  consecutive  days.  The  forecasts  which  were  initiated  based  on 
the  surrounding  initial  conditions  should  imt  also  be  poor  forecasts.  Additionally,  this 
examination  will  determine  if  there  is  a  difference  in  the  dominant  error  pattern  over 
the  CONUS  among  the  three  forecast  lengths,  or  if  the  patterns  are  similar. 

The  4-,  5-  and  6-day  forecast  error  fields  from  one  of  the  subset  cases  is 
presented  in  Fig.  8.  The  valid  date  of  all  three  forecasts  is  the  same  as  that  of  Fig.  7. 
Visual  inspection  of  these  three  forecast  error  fields  reveals  that  there  are  not 
significant  changes  in  the  locations  of  major  otqts  for  the  forecast  valid  date  at  the 
three  forecast  lengths.  Specifically,  the  high/low  error  dipole  near  Baja  and  the 
negative/positive  portion  of  the  positive/negative/positive  error  train  along  the 
Canadian-United  States  border  are  evident  on  all  three  days.  The  ^  tiiat  all  tiuee 
days  produce  errors  in  similar  locations  indicates  that  the  initial  conditioni  are  not  tiiat 
important  in  producing  a  forecast  for  tiiis  valid  date.  There  w^e,  however,  noticeable 
differences  in  the  amplitudes  of  the  error  centers.  This  was  especially  true  of  the  4-day 
errors  over  the  CONUS  itself.  Generally,  the  4-day  forecasts  for  the  valid  dates  in  ti» 
subset  performed  better  than  the  5-  and  6-day  forecasts.  This  trend  was  noted 
throughout  the  entire  sample  subset.  In  fact,  for  tiie  valid  dates  of  poor  S-day 
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forecasts,  S3%  of  the  corresponding  O^iay  ftvecasts  werc  also  deemed  pow,  while  only 
42%  of  the  equivalent  4-day  forecasts  were  poor  performers.  All  three  days  produced 
poor  forecasts  in  only  29.6%  of  the  cases.  Tho^fore,  extoisive  runs  of  poor  forecasts 
do  not  appear  to  be  a  frequent  occurrence. 

C.  Empirical  Orthogonal  FnnctiiMi  analysis  of  errtws 

The  research  presented  thus  fsr  is  limited  by  the  foct  that  it  is  highly  subjective. 
Additionally,  due  to  high  variability  in  500  mb  patterns,  the  appearance  of  a  limited 
number  of  dominant  error  patterns  would  have  been  somewhat  fortuitous.  TherefOTe, 
more  objective  methods  of  analysis  must  be  used  to  reveal  the  underlying  patterns  of 
error  growth.  Empirical  Orthogonal  Function  (EOF)  analysis  is  used  to  gain  objective 
results. 

EOF  analysis  has  become  a  standard  tool  of  exploratory  data  analysis  (Richman 
1986,  Preisendorfer  1988).  This  method  of  analysis  is  used  to  decompose 
mathematically  a  data  set  into  its  dominant  modes,  while  also  determining  the  weight  or 
significance  of  these  modes.  Modes  ate  selected  by  using  matrix  opoatirais  to  compute 
the  eigenvectors  of  a  data  correlation  matrix.  The  eigenvectors  of  this  matrix  define 
the  modes  of  the  data  set,  while  the  accomfxmying  eigoivalues  rqnesent  the  respective 
weights  of  the  eigenvectors.  The  weight  of  the  eigenvectors  represent  the  percentage  of 
the  total  variance  of  the  data  set  explained  by  diat  eigenvector. 

EOF  analysis  is  applied  through  in  the  following  two  imx:edures:  First,  the 
temporal  growth  of  the  RMS  errors  is  analyzed  over  the  length  of  die  forecast  poiod. 
Secondly,  the  spatial  EOFs  of  the  forecast  error  fields  are  computed  fm*  each  day  of  the 
forecast  period.  The  temporal  EOFs  show  the  dominant  patterns  of  error  growdi  rate. 
The  spatial  EOFs  highlight  pattons  and  regions  of  high  and  low  error  growth  rates. 
EOF  decomposition  of  individual  forecast  days  (forecasts  for  day  1,  2,  3,  etc.) 
highlight  the  evolving  patterns  of  error  growth.  One  of  the  interesting  results  is  tibat 
the  dominant  error  patterns  changes  from  shorter  to  longer  forecasts.  In  bodi  the 
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temporal  and  spatial  cases,  comparisons  are  made  between  die  results  derived  from  the 
poor  forecast  cases  and  from  the  good  forecasts.  Such  comparistms  point  out  the 
differences  in  die  two  categories  and  possibly  reveal  a  cause  fw  the  poor  forecasts. 

The  spatial  EOF  analysis  is  restricted  to  the  CONUS  region  of  the  gridded  data. 
This  region  contains  200  grid  points.  To  reduce  computational  demands,  the  number 
of  grid  points  used  in  the  EOF  routine  was  reduced.  The  results  discussed  below  are 
computed  using  SO  grid  points  over  the  CONUS.  The  50  points  refvesent  every  odier 
longitude  and  latitude,  beginning  in  the  nordiwest  comer  of  the  CONUS  region.  This 
results  in  an  effective  grid  point  spacing  of  7.5  degrees  within  the  region.  The  use  of 
this  coarse  spacing  is  still  expected  to  resolve  any  major  patterns,  since  the  major  errors 
in  the  field  have  typical  length  scales  greater  than  7.5  degrees. 

1.  Temporal  EOF  analysis 

As  part  of  the  EOF  analysis  of  RMS  error  growth,  the  mean  growth  curve  of 
the  RMS  values  was  derived  for  all  forecasts,  as  well  as  for  the  135  poor  (5-day  RMS 
errors  greater  than  104.49  m^ers)  and  148  good  (5-day  RMS  mors  less  that  56.4 
meters)  forecast  cases.  Figure  9a  summarizes  the  mean  value  of  RMS  error  as  a 
function  of  forecast  length  for  each  category. 

Inspection  of  the  curves  shows  that  the  RMS  error  growth  rate  of  all  forecasts  is 
essentially  linear,  with  an  increase  of  tdwut  15  meters  per  day  thrmigh  5  days, 
decreasing  to  12  meters  per  day  on  day  6.  This  is  inconsistent  with  the  normal  concept 
of  exponential  growth  rate  and  error  doubling  time.  The  implication  of  this  linear 
growth  rate  is  that  the  errors  are  not,  on  average,  the  results  of  a  simple  linear 
instability  with  a  fixed  error  growth  rate.  However,  neither  the  set  of  poor  or  good 
forecasts  parallels  this  growth  curve.  For  the  collection  of  poor  forecasts,  the  growth  is 
still  linear  for  days  1  and  2,  but  with  a  growth  rate  of  about  20  meto^  per  day. 
However,  after  day  2,  the  growth  curve  increases  in  slope,  with  a  maximum  growth 
rate  of  30  meters  per  day  between  days  3  and  5.  After  this  time,  the  growtii  of  tte 
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Figure  9.  Mean  growdi  of  the  CONUS  RMS  errOT  by  fnecast  day  and  categcny.  (a.) 
Means  RMS  values  and  (b.)  plus  and  minus  one  sigma  for  poor  and  good  forecasts. 
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RMS  enxH*  is  not  as  dramatic.  These  growth  rates  may  owe  dieir  inoiease  over  time  to 
some  kind  of  linear  instability.  This  resuh  indicates  dtat  the  RMS  of  pow  S-day 
forecasts  increases  most  significantly  on  the  synoptic  time  scale.  The  growdi  of  errors 
for  good  forecasts  is  less  dramatic,  as  die  most  growth  (10-14  meters  per  day)  occurs 
by  day  3.  After  this  day,  daily  growth  is  recbiced  to  6  meters  or  less  per  day  durou^ 
day  5.  This  result  shows  that,  while  die  majority  of  RMS  errOT  in  good  forecasts 
occurs  prior  to  day  3,  the  opposite  is  true  for  poor  forecasts,  indicating  that  poor 
forecasts  do  not  become  decidedly  poor  until  midway  through  die  S-day  forecast. 
Additionally,  the  collection  of  poor  forecasts  has  a  much  greato'  variability  (as 
expressed  by  its  standard  deviation  around  the  mean  of  poor  forecasts)  than  die 
collection  of  good  forecasts  (Fig.  %).  The  restricted  variability  of  the  good  forecasts 
effectively  provides  a  lower  bound  on  RMS  performance,  while  the  higher  variability 
of  the  poor  forecast  population  allows  for  much  more  extreme  values  in  this  categmy. 
The  two  forecast  categories  are  essentially  sqiarated  by  day  4  of  the  forecast  period. 
Further,  the  fact  that  the  mean  of  the  13S  poor  forecasts  moves  further  away  from  the 
overall  mean,  and  have  a  greater  variability  than  die  148  poor  forecasts,  is  furdier 
evidence  that  the  overall  distributicm  of  5-day  RMS  values  is  not  a  normal  or  Gaussian 
distribution. 

The  EOF  analysis  of  all  forecasts  for  a  6-day  poiod  yields  eigenvectors  with 
proportions  presented  in  Fig.  10.  This  EOF  analysis  removed  the  mean  error  growdi 
rate.  Therefore,  the  EOFs  explain  only  die  deviations  from  the  mean  mor  growth 
rate.  The  prc^xntions  in  Fig.  10  re^nesent  the  percent  of  variance  explained  by  the 
individual  eigenvectors  or  EOFs.  If  each  EOF  were  to  eiqilain  equal  prqxirtions  of  die 
total  variance,  their  percmitage  would  be  16.6%  (1  part  out  of  6),  since  diere  are  6 
EOFs.  Only  EOFs  1  and  2  (Fig.  1 1)  rqxresent  percentages  gneata-  dian  1  out  of  6  and 
are  deemed  significant.  A  scatter  pl(H  of  the  amplitudes  of  EOF  1  versus  EOF  2  is 
presented  in  Fig.  12.  A  test  was  perframed  using  the  null  hypodiesis  that  die 


Figure  10.  Percentage  of  variance  e}q>lained  by  the  temporal  EOFs  of  CX)NUS  RMS 
error  growth. 
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Figure  1 1 .  First  two  tempcnral  EOFs  of  CONUS  RMS  ertOT  growth. 
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Figure  12.  Scatto^ram  of  the  amplitiules  of  teoqxiral  EOF  1  versus  EOF  2  for  all 
forecasts. 

distribution  in  Fig.  12  is  diffoent  from  a  conq)letely  random  distribution  of  the 
amplitudes.  The  random  distribution  assumes  that  one  fourth  of  the  amplitudes  fall  in 
each  quadrant.  The  p-value  for  diis  test  is  greatm’ dian  .1.  This  p-value  indicates  diat 
die  two  distributions  could  be  selected  at  random  more  than  10%  of  tihe  time. 
Therefore,  die  null  hypothesis  that  die  distribution  is  differmt  than  a  random 
distribution  can  not  be  refected. 

Within  each  quadrant,  in  addition  to  counting  die  total  nundier  of  cases,  the 
number  of  poor  forecasts  was  also  detamined  by  plotting  die  EOF  1  and  EOF  2 
amplitudes  of  just  the  poor  forecasts.  It  was  noted  that  iKiiile  only  21.4%  of  all 
forecasts  woe  found  in  the  quadrant  of  positive  EOF  1  and  negative  EOF  2,  76.6%  of 
the  poOT  5-day  frnecasts  were  located  within  diis  regkMi  (Fig.  13).  The  null  hypodiesis 
that  tins  distribution  is  the  same  as  a  random  distribution  was  tested.  Again  using  the 
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Figure  13.  Scattergram  of  the  amplitudes  of  tempOTal  EOF  1  versus  EOF  2  for  the 
subset  of  poor  forecasts. 


test,  this  result  has  a  p-value  less  than  .001,  representing  a  less  dun  1  in  1000 
probability  of  selecting  such  a  distributicm  at  random.  Therefore,  the  null  hypothesis 
that  die  distribution  of  the  amplitudes  of  EOFs  1  and  2  of  poor  forecasts  is  rgected. 
The  combination  of  positive  EOF  1  and  negative  EOF  2  is  a  significant  pattern  of  RMS 
aTOT  growth  deviati(»  from  the  mean  in  pom-  forecasts. 

Analysis  of  this  combination  of  EOFs  1  and  2  shows  that  EOF  1,  rqneseiiting 
52.8%  of  the  RMS  errOT  variance,  is  relatively  flat.  The  daily  contribution  of  this  EOF 
does  not  vary  gready  from  day  to  day,  althou^  it  does  increase  slightly  through  day  4 
and  then  decreases  somewhat  on  days  S  and  6.  This  decrease  actually  rqiresaits  a 
return  toward  the  mean  RMS  emH-  grov^  rate.  Since  the  EOF  is  relatively  flat,  it 
shows  that  much  of  the  deviation  from  die  mean  growth  rate  is  already  present  on  day 
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1.  The  decrease  in  tile  later  dates  is  due  to  the  teadeocy  of  die  inodd  to  converge  back 
to  its  own  climatology. 

On  the  other  hand,  EOF  2,  becuise  of  its  negative  phase,  initially  represents  a 
tendency  for  pow  forecasts  to  remain  below  or  near  die  mean  curve  fo€  die  first  3  days 
of  die  period.  As  die  fcH-ecast  period  goes  beytmd  days  3  and  4,  the  contributkm  from 
this  EOF  becomes  positive,  indicating  that  poor  forecasts  with  large  negative  EOF  2 
grow  most  rapidly  in  die  laxter  part  of  the  frvecast.  The  change  in  fdiase  of  this  EOF  is 
direcdy  related  to  the  period  of  greatest  RMS  growth  of  the  135  poor  forecasts. 

Conversely,  the  distributimi  of  die  anqilitudes  of  EOFs  1  and  2  for  die  148  good 
frxocasts  was  determined  (Fig.  14).  The  result  was  equally  as  impressive  as  die 
distribution  of  poor  frn'ecasts,  in  that  most  of  the  good  frxecasts  were  cmicentrated  in 
the  opposite  quadrant  from  the  poor  cases.  The  combination  of  negative  EOF  1  and 
positive  EOF  2  contained  28.0%  of  all  forecasts  but  79.9%  of  the  good  frvecasts. 
Again,  the  null  hypothesis  that  this  distnlnitioii  is  the  same  as  a  random  distribution 
was  tested  using  die  ;{^  test.  The  resulting  p-value  is  again  less  than  .(X)l,  indicating 
that  this  is  a  significant  combination  of  EOFs.  In  this  case,  the  opposite  sign  of  EOF  1 
largely  explains  the  deviation  of  the  good  forecasts  to  a  locatira  below  the  mean  errm- 
growth  curve,  as  opposed  to  the  poor  forecasts  aiiich  deviate  to  a  locatira  above  the 
mean  curve.  Additionally,  EOF  2  diffiO’entiates  between  die  rapid  growdi  of  RMS 
error  on  days  3  and  4  of  poor  forecasts  and  die  very  slow  growdi  of  RMS  error  on  the 
same  days  of  the  poor  forecasts.  This  dichotomy  indicates  that  die  departure  frxim  the 
mean  RMS  growth  curve  for  good  forecasts  is  exacdy  the  q^xisite  as  diat  of  die  poor 
forecast  growth. 

While  the  EOF  analysis  presented  above  deimnistrates  die  signifrcam  modes  of 
RMS  errcM'  departure  from  the  mean  errOT  growdi  rate,  it  does  so  at  die  eiqiense  of 
removing  the  actual  mean  growth  pattern.  Because  of  diis,  die  EOF  analysis  was 
recalculated  for  all  forecasts  widiout  removing  the  mean  error  growth  rate.  The 
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Temporal  EOF  1 

Figure  14.  Scattergram  of  the  anqiliPides  of  temporal  EOF  1  versus  EOF  2  for  the 
subset  of  good  forecasts. 


resulting  eigenvectors  are  presented  in  Fig.  IS.  In  this  analysis,  die  mean  pattern  of 
growth  is  dominant  and  the  remaining  eigenvectors  rqnesent  the  departures  from  die 
mean,  similar  to  the  discussion  above.  The  weights  of  the  eigenvectors  for  this  analysis 
are  96.3,  1.7,  0.1,  O.S,  0.3  and  0.1%  reflectively.  These  values  indicate  that  the 
mean  pattern  rqiresents  the  bulk  of  all  RMS  error  variance  and  the  deviations  are  a 
much  smaller  3.7%  of  the  total. 

2.  Spatial  EOF  analysis 

The  research  presented  thus  fta  has  shown  that  the  greatest  amount  of  errOT 
growth  in  poor  5>day  forecasts,  as  judged  by  the  RMS  values  and  their  growth,  occurs 
between  days  3  and  5.  This  correspcmds  to  the  time  period  ftn*  synoptic  scale 
meteorological  inocesses.  In  ordo-  to  d^mmine  if  die  error  fields  of  pom-  and  good  S- 
day  forecasts  are  related  to  synqitic  scale  {xocesses,  these  fields  are  analyzed  fiatially 
via  the  EOF  |HX)cedure.  This  procedure  is  pmformed  to  die  earn  fields  of  each  day  in 
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Figure  15.  Leading  tenqxiral  EOFs  of  CONUS  RMS  eiTOT  growth  after  correcting  for 
mean  growth. 

the  evolving  poor  5-day  forecast.  As  a  result,  the  dominant  evolutionary  patterns 
appear,  and  their  changes  through  the  fraecast  period  (botii  in  spatial  patterns  and 
dominance)  described. 

As  discussed  previously,  tiie  qratial  EOFs  are  computed  using  a  reduced  set  of 
grid  points  (50  vs.  2(X))  within  the  CONUS  region  of  intmest.  On  day  5,  the  resulting 
50  EOFs  explain  portimis  of  the  total  variance  of  pom*  and  good  forecasts  as  shown  in 
Fig.  16.  From  the  graph,  it  is  apparoit  tiiat  ti^  first  few  EOFs  combine  to  erqrlain 
much  more  of  the  total  variance  in  poor  forecasts  tium  in  good  fmecasts.  In  fact,  tiie 
gra{riis  for  days  1-4  (only  day  5  is  shown  in  Fig.  16)  reveal  diat  tiie  dominance 
associated  witii  the  leading  EOFs  increases  witii  time.  Fot  good  fmecasts,  tiiere  are  no 
such  dominant  leading  EOFs.  Since  these  are  already  defined  as  good  forecasts,  it  is 
reasonable  to  expect  that  there  is  not  a  dominant  pattern  of  error;  tiiat  is,  tiie  good 
forecasts  are  n(rt  flawed  by  qiecific  pattmns  of  aixn’  growtii.  As  with  tiie  tmnpoial 
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Figure  16.  Percentage  of  variance  of  the  day  S  CONUS  error  fields  e^lained  by  the 
spatial  EOFs  calculated  from  the  correlation  matrix  for  128  poor  forecasts.  The  dotted 
line  gives  the  avouge  variance  e^qilained  at  each  grid  point,  for  reference. 


analysis,  these  fields  and  the  resulting  EOFs  are  not  conected  for  die  mean  daily  error 
field.  TherefOTe,  the  EOFs  rqnesoit  the  deviatimis  of  the  errOT  field  about  die  mean 
error  field  of  the  reflective  day.  This  should  not  be  a  problem,  since  die  mean  errm*  at 
individual  points  is  close  to  lexo;  that  is,  there  is  only  weak  systematic  bias  of  the 
forecasts  over  the  CONUS  regimi.  The  results  are  first  presented  for  the  fntial  EOF 
analysis  performed  on  the  arm:  fields  of  the  subset  of  poor  fOTecasts. 

The  first  two  EOFs  from  day  1  of  die  poor  forecasts  are  presoited  in  Fig.  17. 
The  pattern  of  EOF  1  is  simply  a  single  phase  with  all  points  of  the  same  sign.  This 
single  phase,  relatively  flat  field  is  similar  to  ^liiat  normally  results  from  EOF  analysis 
of  geophysical  data.  Even  though  diis  field  is  relatively  flat,  there  is  a  weak  maximum 
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b. 


Figure  17.  Leading  correlation  EOFs  of  die  day  1  errOT  field  of  poor  forecasts. 
Contour  interval  is  .05.  Negative  contours  are  dashed. 
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along  the  southeast  border  of  the  region.  The  second  EOF  reveals  a  dipole  pattern 
aligned  northwest  to  southeast  over  the  region.  This  pattern  is  also  fairly  normal  in  die 
context  of  EOF  analysis.  The  scale  of  this  wave  pattern  is  at  least  12,000  km,  or  twice 
the  domain  width.  Together,  these  two  patterns  combine  to  explain  30%  of  the  total 
error  field  variance. 

Analysis  of  the  leading  EOFs  on  day  2  of  the  poor  forecast  runs  (Fig.  18) 
begins  to  reveal  a  systematic  evolution  of  the  error  field.  The  dipole  pattern  which  was 
the  second  EOF  on  day  1  is  now  the  leading  EOF,  although  its  contribution  to 
variance  has  increased  from  10%  to  only  14%.  The  second  EOF  is  essentially  a  single 
phase  pattern,  except  for  one  grid  point  over  Arizona.  This  pattern  is  similar  to  the 
leading  pattern  of  day  1.  Within  this  pattern,  the  earlier  weak  maximum  ova*  the 
Southeast  has  become  more  evident  as  an  axis  of  maximum  amplitude  extending  into 
the  Ohio  Valley.  Additionally,  the  dominance  of  this  pattern  has  been  reduced  by  one 
third,  as  it  is  now  responsible  for  only  12%  of  the  total  variance.  Even  though  the 
same  patterns  exist  on  the  first  two  days,  their  combined  relevance  is  reduced,  while 
they  have  also  switched  positions  with  each  odier.  The  third  EOF  begins  to  reveal  an 
interesting  feature.  This  EOF  is  the  quadrature,  or  90  degree  phase  shift  of  the  first 
EOF.  This  is  evident  since  the  regions  of  maximum  and  minimum  amplitude  of  EOF  1 
are  aligned  along  gradient  regions  of  EOF  3,  and  vice  versa.  The  two  phases  of  these 
two  EOFs  combine  to  reveal  all  four  phases  of  a  standing  wave  pattern,  which  e^qilains 
25%  of  the  variance  of  the  error  fields  on  this  day.  The  switching  of  the  first  two 
EOFs,  along  with  the  iq)pearance  of  the  quadrature  pattern  are  is  indicative  of  the 
changes  underway  in  the  evolution  of  the  forecast  error  fields. 

The  leading  pattons  on  day  3  are  shown  as  Fig.  19.  The  leading  pattern  is  once 
again  the  dipole  over  the  CONUS,  which  has  increased  in  dominance  to  over  19%  and 
become  much  better  resolved.  The  dipole  has  also  lost  much  of  its  ntntfawest  to 
southeast  orientation  and  is  now  aligned  almost  zonally.  The  quadrature  pattern  is  now 
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Figure  18.  Leading  correlation  EOFs  of  the  day  2  error  field  of  poor  forecasts. 


POOR  FORECASTS 


Figure  18.  (Continued). 


a. 


b. 

Figure  19.  Leading  correlation  EOFs  of  the  day  3  error  field  of  poor  forecasts. 


46 


Figure  19.  (Continued). 


the  second  EOF,  and  the  combination  of  diese  two  patterns  increases  in  dominance  to 
32%.  Thus,  this  wave  patton  now  explains  about  one  tiliird  of  the  total  variance.  The 
single  phase  pattern  which  was  present  in  die  first  two  days  is  not  as  readily  iqjparent 
by  day  3.  It  might  be  found  most  closely  in  EOF  3,  althougih  diis  pattern  has  a  greater 
negative  area  than  was  found  in  EOF  2  on  day  2.  Additionally,  since  EOFs  2  and  3 
eiqilain  nearly  the  same  percentage  of  the  variance  (12.86%  vo^  11.05%),  there  is 
some  mixing  of  signal  between  the  two. 

The  leading  EOFs  of  day  4  (Fig.  20)  indicate  that  the  combination  of  patterns 
which  emerged  on  day  3  have  ccmtinued  to  do  so.  The  dipole  and  its  quadrature  have 
separated  from  die  remaining  patterns  and  now  combine  to  explain  40%  of  the  total 
variance  in  die  error  fields.  These  two  continue  to  be  die  leading  modes  on  day  5  (Fig. 
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a. 


b. 

Figure  20.  Leading  correlation  EOFs  of  the  day  4  error  field  of  po(»r  frarecasts. 
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21),  with  similar  c(HitributiQiis  to  the  variance  of  the  enm  fields.  The  single  phase 
pattern  from  days  1  and  2  is  found  among  the  first  5  EOFs  on  eidier  day,  indicating 
that  it  no  Icmger  contributes  strongly  to  the  error  fields. 

The  evolution  of  EOFs  for  the  set  of  good  forecasts  takes  a  different  course  and 
reveals  their  differences  from  the  EOFs  of  the  poor  forecasts.  On  day  1  (Fig.  22),  die 
leading  pattern  is  nearly  identical  to  diat  of  the  poor  fraecasts:  a  single  phase, 
relatively  flat  field  which  explains  20%  of  the  total  variance.  However,  die  dipolar 
pattern  of  EOF  2,  while  being  somewhat  weako*,  has  a  mmre  (vonounced  northwest  to 
soudieast  orientation  than  did  this  pattern  in  poor  forecasts.  On  day  2  (Fig.  23),  die 
two  patterns  switch  dominance,  as  they  did  in  poOT  forecasts.  Additionally,  die  wave 
pattern  for  good  forecasts  is  in  quadrature  with  the  leading  wave  pattern  of  the  poor 
forecasts.  However,  the  closeness  of  this  arntribution  to  variance  by  EOFs  1  and  2, 
both  in  relation  to  each  other  and  to  the  remaining  EOFs,  reveal  that  neither  is  a  very 
dominant  feature,  as  in  poor  forecasts.  The  mixing  of  diese  patterns  is  further 
evidenced  by  days  4  and  S,  when  diey  once  again  switch  positions  (Figs.  24  and  2S). 
Finally,  it  is  noted  that  the  quadrature  pattern,  which  was  clearly  evidoit  in  the  poor 
forecasts,  is  not  as  dominant  among  the  first  5  EOFs  of  any  day  in  die  good  forecast 
subset. 

The  EOF  analysis  as  discussed  above  was  perfonned  by  analyzing  the 
correlation  matrix  of  the  error  fields.  As  a  result,  die  pattens  which  are  revealed  have 
been  smoothed  by  the  standard  deviations  of  the  individual  points  in  the  oror  field. 
The  EOF  analysis  was  once  p^formed  again,  this  time  using  the  values  contained  in 
the  covariance  matrix  or  the  error  fields.  The  resulting  EOFs  from  diis  analysis  are 
patterns  which  additimially  emphasize  the  variance  of  die  forecast  otot  at  all  individual 
points  in  the  errOT  fields.  Some  of  the  results  from  diis  second  analysis  will  be 
discussed  below  and  compared  to  the  previous  results. 
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Figure  21.  Leading  correlati(Hi  EOFs  of  the  day  S  error  field  of  poor  forecasts. 
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Figure  23.  Leading  cmrelation  EOFs  of  the  day  2  error  field  of  good  fcnecasts. 
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Figure  24.  Leading  correlaticHi  EOFs  of  the  day  4  oror  field  of  good  f^ecasts. 
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a. 


b. 

Figure  25.  Leading  correlation  EOFs  of  the  day  5  errcn*  field  of  good  fm-ecasts. 
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On  day  1  (Fig.  26),  the  leading  EOF  is  once  again  the  single  frfiase  pattern  as 
before,  and  as  expected.  However,  the  contribution  to  variance  is  mcaie  pixmounced. 
The  second  EOF  still  has  some  charactoistics  of  the  dipole  patton,  but  its  existence  is 
largely  confined  to  the  western  half  of  the  CX)NUS,  as  opposed  to  the  longer  lengdi 
scale  found  in  the  covariance  EOFs.  The  single  iduuse  remains  dominant  (m  day  2  (Fig. 
27a),  however  the  presence  of  a  maximum  centm*  in  Washington  resembles  die  dqxilar 
pattern.  The  dipole  pattern  and  its  quadrature  is  revealed  in  EOFs  2  and  3,  however 
the  two  modes  are  not  well  separated,  either  from  each  other  or  from  the  remaining 
EOFs,  in  terms  of  dominance  (Fig.  27b  and  27c).  On  day  3  (not  shown)  die  dipole  and 
quadrature  gain  dominance  as  before,  with  diis  dominance  continuing  through  day  5 
(Fig.  28).  Once  again,  it  must  be  noted  diat  when  this  analysis  is  performed,  the 
significance  of  the  dipole/quadrature  pattern  is  shown  to  be  increased. 

For  good  forecasts,  the  single  phase  pattern  is  once  again  the  leading  pattern  on  day 
1  (Fig.  29).  However,  throughout  the  l«igth  of  the  forecast,  the  dipole/quadrature 
combination  still  never  manifests  itself  as  oa&  of  die  dominant,  leading  patterns.  The 
pattern  is  found  on  day  5  as  EOFs  2  and  3  (Fig.  30),  but  with  a  low,  nondescript 
percentage  of  the  variance.  As  with  the  correlation  EOFs,  the  contributions  to  variance 
of  the  good  forecast  covariance  EOFs  are  muddled  as  revealed  by  their  flat  distribution 
in  Fig.  31. 

The  EOFs  presented  thus  far  are  calculated  without  correcting  for,  or  removing,  die 
mean  error  field  of  each  individual  forecast  day  and  forecast  category,  especially  the 
poor  forecast  category.  To  complete  the  EOF  analysis,  the  EOFs  are  again  calculated 
with  the  mean  error  field  removed.  This  mean  pattern  will  be  revealed  as  the  leading 
mode,  as  was  the  case  of  the  temporal  EOF  presented  earlier.  In  this  case,  die  mean 
pattern  would  represent  the  systematic  model  error  associated  with  the  particular 
forecast  day  and  category. 
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Figure  26.  Leading  covariance  EOFs  of  the  day  1  error  field  of  poOT  forecasts. 
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Figure  27.  Leading  covariance  EOFs  of  the  day  2  error  fields  of  poOT  forecasts. 
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Figure  28.  Leading  covariance  EOFs  of  die  day  3  error  field  of  poor  forecasts. 


Figure  30.  Leading  covariance  EOFs  of  the  day  5  error  field  of  good  forecasts. 


Figure  30.  (Continued). 
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Figure  31.  Percentage  of  variance  of  the  day  5  CONUS  error  fields  explained  by  the 
spatial  EOFs  calculated  from  the  covariance  matrix. 
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When  this  analysis  is  accomplished,  an  interesting  resuh  is  revealed  in  the  poor 
forecasts.  The  mean  pattern  is  essentially  a  zero  field,  and  is  masked  in  the  leading  EOF. 
Inspection  of  the  leading  day  S  poor  forecast  EOFs  (Fig.  32)  reveals  that  these  EOFs  are 
essentially  the  same  as  those  which  were  not  corrected  for  the  mean.  This  similarity 
extends  even  to  the  percentage  of  variance  explained  by  each  EOF.  The  absence  of  an 
identifiable  mean  pattern  through  this  analysis  indicates  that  the  errors  in  poor  forecasts  at 
5  days  do  not  contain  a  systematic  bias. 

As  discussed  earlier,  the  fiict  that  RMS  performance  increased  with  time  over  the 
data  period,  coupled  with  the  use  of  10-year  statistics  to  define  forecast  categories, 
introduced  some  bias  into  the  poor  and  good  forecast  data  sets.  The  poor  forecast  data 
set  is  weighted  more  toward  the  earlier  years  of  the  data  period,  while  the  good  forecasts 
are  slanted  more  toward  the  later  years.  This  means  that  there  may  be  more  contribution 
to  the  error  modes  in  poor  forecasts  fi’om  the  earlier  version  of  the  ECMWF  model.  To 
investigate  if  this  is  indeed  a  factor,  the  EOF  analyris  is  once  again  performed  for  each 
forecast  category  and  length.  However,  in  this  analysis  the  error  fields  are  also  segregated 
on  the  basis  of  the  operational  version  of  the  model  used  to  produce  the  forecast.  In  this 
data  set,  the  winter  seasons  1981-1983  represent  the  grid  point  version,  1984  and  1985 
are  from  the  T63  version  and  the  1986-1990  winter  seasons  are  fi’om  the  T106  model. 

Figure  33  presents  the  leading  correlation  EOFs  for  day  5  of  poor  forecasts  from 
the  1981-1983  winter  seasons.  During  this  period,  the  grid  point  version  of  the  model  was 
the  operational  version.  The  familiar  dipole  pattern  is  once  again  the  leading  EOF  at  this 
forecast  length  and  category.  This  mode  contributes  27%  of  the  variance  in  this  case. 
The  quadrature  pattern  is  once  again  revealed  in  EOFs  2  and  3.  These  two  modes  are  also 
somewhat  muddled,  as  their  contributions  are  rather  close  (IS  and  13.6%,  respectively), 
although  EOF  2  looks  most  like  the  quadrature.  Compare  these  patterns  with  the  EOFs  of 
Fig.  21 .  This  comparison  shows  that  the  dominant  modes  in  this  category  fi'om  this  3  year 
period  are  essentially  the  same  as  those  of  the  entire  10-year  period. 
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b. 


Figure  32.  Leading  correlation  EOFs  with  mean  error  field  included  of  die  day  5  oror 
fields  of  poor  forecasts. 
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b. 


Figure  33.  Leading  correlation  EOFs  of  the  day  5  error  field  of  poor  forecasts  fi’om  the 
operational  grid  point  version  of  the  ECMWF  model. 
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The  leading  EOFs  from  the  day  5  error  field  of  poor  forecasts  from  the  T106 
version  of  the  ECMWF  are  presented  in  Fig.  34.  Once  again,  the  dipole  pattern  is  the 
leading  pattern,  explaining  about  31.5%  of  the  variance.  The  quadrature  pattern  again 
takes  its  place  as  EOF  2.  This  dipole/quadrature  pattern  is  now  revealed  as  the  leading 
mode  in  the  error  fields  of  poor  forecasts  of  the  grid  point  model  and  the  T106  model,  as 
well  as  the  entire  10-year  set  of  poor  forecasts.  Earlier  in  Chapter  V,  the  inherait  bias 
caused  by  the  use  of  10-year  statistics  was  discussed.  This  resulted  in  the  sdection  of 
more  poor  forecasts  fiom  the  earlier  years  of  the  10-year  period,  less  poor  forecasts 
selected  from  the  later  years  of  the  data  set.  However,  there  does  not  appear  to  be  a 
serious  drawback  in  using  the  10-year  statistics  to  differentiate  betweoi  poor  and  good 


Figure  34.  Leading  correlation  EOFs  of  the  day  5  error  field  of  poor  forecasts  fi'om  the 
T106  operational  version  of  the  ECMWF  model. 
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forecasts,  since  all  three  of  th^  data  sets  exhibit  the  same  aixM-  modes  in  poor  fixecasts 
on  day  S. 

3.  Summary 

The  existence  of  this  quadrature  pattern  in  poor  forecasts,  but  not  good  forecasts, 
requires  further  discus»on.  The  leacUng  dipole  pattern,  which  is  centa«d  around  the 
Rockies,  resembles  that  discussed  by  Aipe  and  Klinker  (1986).  Howeva*,  the  existence  of 
its  quadrature  is  important  also.  When  both  phases  (positive  and  native)  of  these  two 
EOFs  are  taken  in  combination,  the  result  is  all  four  phases  of  what  appears  to  be  a 
standing  wave  originating  due  to  the  Rocky  Mountains.  The  oscillation  of  this  wave  is 
controlled  by  the  daily  amplitudes  of  the  respective  EOFs.  When  the  pattern  is  most 
influenced  by  the  leading  EOF,  the  amplitude  nuudma  of  the  resulting  error  field  will  most 
resemble  the  dipole  pattern  However,  when  the  quadrature  portion  contributes  more  to 
the  total  pattern,  the  amplitude  maxima  of  the  resulting  error  field  will  be  shifted  further 
downstream  from  the  Rockies.  This  apparent  movement  of  this  feature  develops  on  the  3 
-  S  day  time  scale.  This  time  scale  was  also  identified  by  the  temporal  EOF  analysis  as  the 
time  of  maximum  RMS  growth  in  poor  forecasts.  Thus,  it  appears  that  the  conditions  and 
times  of  ma^um  RMS  growth  are  directly  related  to  tte  appearance  of  this  orographic 
feature. 

This  apparent  standing  wave  over  the  CONUS  in  the  error  fields  of  poor  forecasts 
is  tested  to  determine  if  it  can  be  associated  directly  with  the  occurrence  of  poor  forecasts. 
Specifically,  it  is  speculated  that  these  two  EOFs  reveal  an  orogr^hic  instability  in  the 
form  of  a  standing  downstream  wave  resulting  fi-om  the  Rockies  and  that  this  wave  can  be 
related  to  the  mean  zonal  wind  over  the  CONUS. 

Zonal  mean  500  mb  heights  over  the  sector  of  the  CONUS  region  are  computed  at 
two  latitudes  (30  N  and  48.75  N)  for  use  in  the  geostrophic  wind  equation  of  Holton 
(1992).  These  ^ve  a  value  of  the  mean  500  mb  zonal  wind  between  approximately  30  and 
50  N.  The  zonal  winds  were  computed  for  both  the  initial  conditions  of  all  forecast  runs. 
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as  well  as  for  each  forecast  day.  The  values  of  5-day  CONUS  RMS  were  then  plotted 
against  the  zonal  wind  speeds  of  the  corresponding  initial  conditions  which  produced  the 
forecasts.  The  results  are  shown  in  Fig.  35.  If  the  poor  forecasts  were  related,  via  the 
zonal  wind,  to  the  standing  wave  orographic  instability,  the  RMS  values  of  poor  forecasts 
should  be  restricted  to  a  narrow  range  of  zonal  wind  speeds.  This  range  should  also  be 
exclusive  of  the  wind  speeds  related  to  the  other  categories.  However,  this  separation  of 
good  and  poor  forecasts  on  the  basis  of  mean  zonal  winds  was  not  found  to  be  the  case,  as 
the  distributions  of  mean  zonal  wind  speeds  are  essentially  equivalent  for  both  poor 
forecasts  and  good  forecasts  (Fig.  36).  Similar  poor  resuhs  were  obtained  when  this 
comparison  was  made  using  the  mean  zonal  winds  computed  over  several  other 
geographical  regions  immediately  upstream  from  the  CONUS  region. 
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Figure  35.  Scattergram  of  the  relationship  between  the  initial  condition  mean  zonal  wind 
speed  and  the  5-day  CONUS  RMS  value. 
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CHAPTER  VII 
SUMMARY 


The  error  fields  of  poor  S-day  CONUS  forecasts  have  been  examined  in  three 
ways.  These  examinations  have  revealed  several  interesting  features  about  die  growdi 
of  errors  in  these  forecasts,  both  from  temporal  and  spatial  points  of  view. 

Examination  of  the  raw  error  data  revealed  that  the  CONUS  region  is  in  a 
relative  minimum  area  when  measuring  the  frequency  of  errors  at  the  5-day  point.  The 
maxima  which  lie  just  upstream  and  downstream  of  the  CONUS  are  in  documented 
areas  of  persistent  anomalies.  The  errors  over  die  CONUS  are  fairly  evenly  divided 
between  under  and  over  forecasts  of  the  S-day  SOO  mb  height. 

The  RMS  error  value  for  the  CONUS  was  used  as  the  measure  of  forecast 
performance  over  the  region  of  interest.  In  keeping  with  previous  woik  with  this  data 
set,  a  normal  distribution  of  this  value  was  assumed.  However,  in  this  study,  the  10- 
year  mean  and  standard  deviation  were  used  to  provide  critical  values  for  poor  and 
good  forecasts.  This  was  done  to  avoid  the  paradox  in  which  overall  forecast 
performance  improves,  (as  measured  by  annual  RMS  values),  but  the  actual  number  of 
poor  forecasts  increased  with  time.  However,  this  creates  a  bias  in  the  data  due  to  the 
general  decrease  of  the  RMS  error  with  time.  The  subset  of  poor  forecasts  is  weighted 
more  toward  tiie  early  years  of  the  period,  while  the  subset  of  good  forecasts  is  slanted 
more  toward  the  latter  portion  of  the  period.  A  similar  study  of  the  RMS  distribution 
was  accomplished  for  the  East  Pacific  Ocean  region  immediately  west  of  the  CONUS 
region.  It  was  found  that  the  distribution  of  5-day  RMS  values  for  this  region  was  not 
significantly  different  than  that  over  the  CONUS. 

A  subjective  analysis  of  the  error  growth  of  a  subset  of  the  poor  forecasts  was 
accomplished.  This  revealed  no  outstanding  pattern  of  raw  error  growth,  location  of 
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error  initiation  zones  or  movement  of  the  error  action  centers  with  time.  In  fact, 
maximum  error  amplitudes  in  the  poor  forecasts  w^  found  to  occur  throughout  the 
entire  Northern  Hemisphere,  rather  than  in  distinct  regions.  It  was  also  observed  that 
the  error  action  centers  exhibited  very  little  transient  behavior,  but  instead  seemed  to 
grow  in  situ  with  time. 

Additionally,  the  same  subset  of  poor  forecasts  was  analyzed  to  d^ermine  the 
differences  between  forecasts  origiiuiting  on  successive  days  which  verify  on  die  same 
day  four  to  six  days  later.  Again,  for  the  successive  forecasts,  the  main  error  centers 
for  the  CONUS  did  not  exhibit  significant  day  to  day  movement.  A  troid  was  noted 
throughout  the  entire  forecast  data  set  for  die  percentage  of  poor  forecasts  for  these 
valid  dates  to  increase  with  increasing  forecast  length.  This  would  indicate  that  die 
distribution  of  the  RMS  error  value  becomes  skewed  more  with  time.  On  the  odier 
hand,  S-day  forecasts  that  were  poor  were  likely  to  be  poor  6-day  forecasts  started  a 
day  earlier  or  poor  4-day  forecasts  started  a  day  later.  This  result  is  not  consistent, 
however,  with  the  observation  that  poor  forecasts  tend  not  to  be  episodic. 

Due  to  the  limited  success  of  these  subjective  analyses,  a  more  objective 
approach  was  taken  through  the  use  of  EOF  analysis,  both  in  time  and  space.  The 
temporal  analysis  used  the  daily  CONUS  RMS  errors  which  evolved  out  of  each  5-day 
forecast  run.  This  procedure  showed  that  there  were  quantifiable  diffoiences  in  the 
evolution  of  the  RMS  errors  between  poor  and  good  forecasts.  Good  forecasts  were 
found  to  have  the  majority  of  their  RMS  growth  on  the  first  day  of  the  forecast  run. 
However,  poor  forecasts  were  found  to  perform  as  well  as  all  other  forecasts  for  one  or 
two  days  before  experiencing  a  rapid  growth  of  error  on  days  3  and  4.  With  regards  to 
the  dominant  EOFs,  the  paths  taken  by  poor  and  good  forecasts  are  completely 
opposite.  In  particular,  good  forecasts  e^qierienced  excessively  weak  error  growth  on 
days  3-5.  The  dominant  EOFs  were  found  to  explain  a  significant  majority  of  the 
performance  of  both  poor  and  good  forecast  RMS  growth. 
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The  spatial  EOF  analysis  of  Ae  daily  ernM*  fields  of  evolving  pow  and  good 
forecasts  revealed  significant  differences  between  the  two  categ(Hies.  Initially,  both 
categories  were  found  to  exhibit  universally  typical  EOF  modes.  Howeva*,  pow 
forecasts  exhibited  a  pair  of  EOF  modes  which  combine  to  define  an  ^jparrat  standing 
wave  which  seems  to  originate  due  to  the  presence  of  the  Rocky  Mountains.  This 
combination  of  EOFs  becomes  dominant  on  the  3  -  4  day  time  Imgth  indicative  of 
synoptic  weather  systems.  By  day  5,  the  two  EOFs  combine  to  explain  over  40%  of 
the  variance  in  the  error  fields.  No  similar  pattern  or  combination  of  patterns  was 
found  within  the  analysis  of  good  forecasts. 

The  pattern  evidenced  by  the  leading  EOFs  of  poor  forecasts  resembles  a 
standing  wave  which  may  represent  an  orographic  instability.  In  an  attempt  to  relate 
the  performance  of  poor  forecasts  to  the  presence  of  this  wave,  the  RMS  value  of  the 
poor  S-day  forecast  was  compared  to  the  mean  zonal  wind  speeds  over  the  CONUS. 
This  comparison  was  made  using  both  Utit  mean  zonal  wind  from  the  initial  conditions 
and  the  5-day  forecast.  No  tendency  for  poor  forecasts  to  be  associated  with  a 
restricted  range  of  zonal  wind  was  found.  The  same  poor  results  were  found  when  the 
mean  zonal  winds  were  computed  over  other  geographical  regions  upstream  of  the 
CONUS. 

Many  questions  remain  unanswered  regarding  the  causes  of  poor  forecasts,  not 
only  over  the  CONUS,  but  on  a  global  scale.  This  study  has  focused  on  diagnosing  the 
patterns  which  exist  in  the  error  fields  over  the  CONUS  region.  Further  investigation 
is  necessary  to  determiiie  why  these  patterns  produce  poor  forecasts.  It  is  possible  that 
the  leading  EOFs  and  the  poor  RMS  error  values  are  more  closely  related  to  the  wind 
fields  at  other  levels  of  the  atmosphere,  or  even  to  some  combination  of  levels. 
Additionally,  it  has  been  suggested  that  the  error  action  centers  or  the  error  trains  may 
be  related  to  other  dynamic  variables  such  as  the  vorticity  field.  Since  the  leading 
patterns  over  the  CONUS  appear  to  be  linked  to  the  Rocky  Mountains,  refinements  to 
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the  model  orography  in  this  region  may  also  yield  successful  results.  Many  (rther 
possibilities  exist  and  warrant  further  research. 

The  objectives  of  this  research  have  been  met.  Tte  10-year  climatological 
distributions  of  the  S-day  RMS  values  have  been  examined  over  different  geogr^jhical 
regions  in  order  to  define  poor  and  good  f^ecasts.  A  case  study  of  oror  evolutum  was 
presented.  The  growth  of  CONUS  RMS  with  time  has  been  analyzed  using  EOF 
analysis  and  was  found  to  be  different  for  poor  and  good  forecasts.  The  ^Mtial 
evolution  of  the  CONUS  error  fields  was  investigated  with  EOFs  and  the  dominant 
patterns  were  found.  The  leading  spatial  patterns  were  also  found  to  be  different  for 
poor  and  good  forecasts.  These  EOF  results  were  compared  for  different  time  periods 
in  the  evolution  of  the  ECMWF  model  and  no  significant  differences  woe  found.  The 
spatial  patterns  found  in  poor  forecasts  suggested  a  dynamic-orogr^hic  relation 
between  the  poor  forecasts  and  zonal  winds.  However,  no  strong  relation  between  the 
two  values  was  found. 


APPENDIX  A 


VALID  DATES  OF  POOR  FORECASTS 


The  subset  of  poor  forecasts  used  in  this  study  consists  of  S-day  forecasts  valid 
the  following  dates: 


801205 

820125 

840205 

870207 

801206 

820203 

840223 

870208 

801210 

820204 

841206 

870209 

801217 

820210 

841207 

870224 

810102 

820212 

841214 

871229 

810103 

820213 

841222 

871230 

810104 

820214 

841223 

880110 

810113 

820216 

841224 

880124 

810119 

820218 

841228 

880228 

810122 

820220 

850101 

880229 

810125 

820221 

850113 

881212 

810201 

821210 

850116 

881218 

810203 

821211 

850117 

890103 

810209 

821222 

850119 

890108 

810211 

821223 

850122 

890130 

810214 

821228 

850201 

890131 

810215 

821230 

850202 

890201 

810216 

830103 

850209 

890202 

810224 

830110 

850211 

890203 

810225 

830111 

850224 

890228 

810226 

830129 

850308 

890303 

810227 

830202 

851224 

900101 

811221 

830219 

851225 

900112 

820104 

830223 

851227 

900118 

820108 

831215 

851231 

900126 

820109 

831221 

860106 

900301 

820110 

831222 

860131 

900302 

820111 

831223 

860306 

900303 

820115 

831224 

860307 

900304 

820118 

831225 

861207 

820120 

831226 

861219 

820121 

840117 

870123 

820124 

840204 

870206 
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APPENDIX  B 

VALID  DATES  OF  GOOD  FORECASTS 


The  subset  of  good  f(x«casts  used  in  this  study  consists  of  5-day  forecasts  valid 
on  the  following  dates: 


801207 

840123 

860217 

880121 

801208 

840211 

860218 

880129 

801220 

840224 

860219 

880130 

801223 

840303 

860303 

880131 

801224 

840304 

860309 

880205 

810112 

840305 

861205 

880208 

810114 

850126 

861210 

880214 

8101 IS 

850127 

861215 

880216 

810124 

850128 

861220 

880226 

811215 

850215 

861221 

880227 

820129 

850217 

861224 

880306 

820303 

850218 

870108 

880307 

820305 

850225 

870119 

881216 

830107 

850226 

870120 

890105 

830108 

850227 

870128 

890109 

830121 

851209 

870129 

890114 

830122 

851215 

870216 

890117 

830131 

851216 

870217 

890119 

830205 

851217 

870220 

890120 

830208 

851221 

870307 

890128 

830209 

851228 

871207 

890207 

830211 

860102 

871208 

890210 

830301 

860103 

871209 

890211 

830302 

860110 

871212 

890222 

831205 

860121 

871220 

890224 

831206 

860124 

880108 

891207 

840103 

860201 

880112 

891210 

840105 

860205 

880116 

891215 

840107 

860208 

880118 

891219 

840118 

860213 

880119 

891220 

840122 

860215 

880120 

891223 

90010S 

900106 

900109 

900111 

900114 

900115 

900116 

900124 

900128 

900129 

900201 

900206 

900207 

900208 

900211 

900214 

900215 

900225 

903226 

900309 
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